ebab7d0da53c51438720d60c6e2e467c - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Contrastive Language-Image Pre-training</title>

</head>

Contrastive Language-Image Pre-training

</h1>

</header>

/* start https://en.wikipedia.org/ */

.mw-parser-output .infobox-subbox{padding:0;border:none;margin:-3px;width:auto;min-width:100%;font-size:100%;clear:none;float:none;background-color:transparent}.mw-parser-output .infobox-3cols-child{margin:auto}.mw-parser-output .infobox .navbar{font-size:100%}@media screen{html.skin-theme-clientpref-night .mw-parser-output .infobox-full-data:not(.notheme)>div:not(.notheme)[style]{background:#1f1f23!important;color:#f8f9fa}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .infobox-full-data:not(.notheme)>div:not(.notheme)[style]{background:#1f1f23!important;color:#f8f9fa}}@media(min-width:640px){body.skin--responsive .mw-parser-output .infobox-table{display:table!important}body.skin--responsive .mw-parser-output .infobox-table>caption{display:table-caption!important}body.skin--responsive .mw-parser-output .infobox-table>tbody{display:table-row-group}body.skin--responsive .mw-parser-output .infobox-table th,body.skin--responsive .mw-parser-output .infobox-table td{padding-left:inherit;padding-right:inherit}}

/* end https://en.wikipedia.org/ */

</style><table class="infobox vevent"><tbody><tr><th colspan="2" class="infobox-above summary">CLIP</th></tr><tr><td colspan="2" class="infobox-image logo"></td></tr><tr><th scope="row" class="infobox-label" style="white-space: nowrap;"><a href="Programmer" title="Programmer">Developer(s)</a></th><td class="infobox-data"><a href="OpenAI" title="OpenAI">OpenAI</a></td></tr><tr><th scope="row" class="infobox-label" style="white-space: nowrap;">Initial release</th><td class="infobox-data">January 5, 2021</td></tr><tr><th scope="row" class="infobox-label" style="white-space: nowrap;"><a href="Repository_(version_control)" title="Repository (version control)">Repository</a></th><td class="infobox-data"><a rel="nofollow" class="external text" href="https://github.com/OpenAI/CLIP">github.com/OpenAI/CLIP</a></td></tr><tr><th scope="row" class="infobox-label" style="white-space: nowrap;">Written in</th><td class="infobox-data"><a href="Python_(programming_language)" title="Python (programming language)">Python</a></td></tr><tr><th scope="row" class="infobox-label" style="white-space: nowrap;"><a href="Software_license" title="Software license">License</a></th><td class="infobox-data"><a href="MIT_License" title="MIT License">MIT License</a></td></tr><tr><th scope="row" class="infobox-label" style="white-space: nowrap;">Website</th><td class="infobox-data"><a rel="nofollow" class="external text" href="https://openai.com/research/clip">openai.com/research/clip</a></td></tr></tbody></table>

Contrastive Language-Image Pre-training (CLIP) is a technique for training a pair of <a href="Artificial_neural_network" class="mw-redirect" title="Artificial neural network">neural network</a> models, one for image understanding and one for text understanding, using a <a href="Contrastive_learning" class="mw-redirect" title="Contrastive learning">contrastive</a> objective.<a href="#cite_note-:0-1">[1]</a>

This method has enabled broad applications across multiple domains, including cross-modal retrieval,<a href="#cite_note-2">[2]</a> text-to-image generation,<a href="#cite_note-stable-diffusion-github-3">[3]</a> and aesthetic ranking.<a href="#cite_note-4">[4]</a>

<div class="mw-heading mw-heading2"><h2 id="Algorithm">Algorithm</h2></div>

The CLIP method trains a pair of models contrastively.<a href="#cite_note-:0-1">[1]</a> One model takes in a piece of text as input and outputs a single vector representing its semantic content. The other model takes in an image and similarly outputs a single vector representing its visual content. The models are trained so that the vectors corresponding to semantically similar text-image pairs are close together in the shared vector space, while those corresponding to dissimilar pairs are far apart.<a href="#cite_note-:0-1">[1]</a>

To train a pair of CLIP models, one would start by preparing a large dataset of image-caption pairs. During training, the models are presented with batches of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle N}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N}</annotation>

</semantics>

</math><img src="./f5e3890c981ae85503089652feb48b191b57aae3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.064ex; height:2.176ex;" alt="{\displaystyle N}" loading="lazy"> image-caption pairs. Let the outputs from the text and image models be respectively <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle v_{1},...,v_{N},w_{1},...,w_{N}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle v_{1},...,v_{N},w_{1},...,w_{N}}</annotation>

</semantics>

</math><img src="./182a9d6a14eccf955f18bf86696758fd0bd849ec.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:22.448ex; height:2.009ex;" alt="{\displaystyle v_{1},...,v_{N},w_{1},...,w_{N}}" loading="lazy">. Two vectors are considered "similar" if their dot product is large.

The loss incurred on this batch is the multi-class N-pair loss,<a href="#cite_note-5">[5]</a> which is a symmetric <a href="Cross_entropy_loss" class="mw-redirect" title="Cross entropy loss">cross-entropy loss</a> over similarity scores:<math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle -{\frac {1}{N}}\sum _{i}\ln {\frac {e^{v_{i}\cdot w_{i}/T}}{\sum _{j}e^{v_{i}\cdot w_{j}/T}}}-{\frac {1}{N}}\sum _{j}\ln {\frac {e^{v_{j}\cdot w_{j}/T}}{\sum _{i}e^{v_{i}\cdot w_{j}/T}}}}">

<mfrac>

</mfrac>

</mrow>

</mrow>

</munder>

<mfrac>

<msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

<mrow>

</mrow>

</munder>

<msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

<mfrac>

</mfrac>

</mrow>

</mrow>

</munder>

<mfrac>

<msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

<mrow>

</mrow>

</munder>

<msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle -{\frac {1}{N}}\sum _{i}\ln {\frac {e^{v_{i}\cdot w_{i}/T}}{\sum _{j}e^{v_{i}\cdot w_{j}/T}}}-{\frac {1}{N}}\sum _{j}\ln {\frac {e^{v_{j}\cdot w_{j}/T}}{\sum _{i}e^{v_{i}\cdot w_{j}/T}}}}</annotation>

</semantics>

</math>In essence, this loss function encourages the dot product between matching image and text vectors (<math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle v_{i}\cdot w_{i}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle v_{i}\cdot w_{i}}</annotation>

</semantics>

</math><img src="./9a1ef9d1cb5b94b6dce3a0f0b6d6582391ddb2b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:6.07ex; height:2.009ex;" alt="{\displaystyle v_{i}\cdot w_{i}}" loading="lazy">) to be high, while discouraging high dot products between non-matching pairs. The parameter <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle T>0}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle T>0}</annotation>

</semantics>

</math><img src="./e39e90b9e1e7d5be3b5eae57729dc63494bbe3fd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.897ex; height:2.176ex;" alt="{\displaystyle T>0}" loading="lazy"> is the <a href="Temperature" title="Temperature">temperature</a>, which is parameterized in the original CLIP model as <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle T=e^{-\tau }}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle T=e^{-\tau }}</annotation>

</semantics>

</math><img src="./a44639f7aa4f80e75bd3e05a8a964570681924a7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:8.179ex; height:2.509ex;" alt="{\displaystyle T=e^{-\tau }}" loading="lazy"> where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \tau \in \mathbb {R} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau \in \mathbb {R} }</annotation>

</semantics>

</math><img src="./ae9dffeaac885108c5db356c1013f5d47fa22e11.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.721ex; height:2.176ex;" alt="{\displaystyle \tau \in \mathbb {R} }" loading="lazy"> is a learned parameter.

Other loss functions are possible. For example, Sigmoid CLIP (SigLIP)<a href="#cite_note-:4-6">[6]</a> proposes the following loss function:<math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L={\frac {1}{N}}\sum _{i,j\in 1:N}f((2\delta _{i,j}-1)(e^{\tau }w_{i}\cdot v_{j}+b))}">

<mfrac>

</mfrac>

</mrow>

</mrow>

</munder>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L={\frac {1}{N}}\sum _{i,j\in 1:N}f((2\delta _{i,j}-1)(e^{\tau }w_{i}\cdot v_{j}+b))}</annotation>

</semantics>

</math>where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f(x)=\ln(1+e^{-x})}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f(x)=\ln(1+e^{-x})}</annotation>

</semantics>

</math><img src="./8c86c2a2542256830d0512f51b55678717accd53.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:18.802ex; height:3.009ex;" alt="{\displaystyle f(x)=\ln(1+e^{-x})}" loading="lazy"> is the negative log <a href="Sigmoid_function" title="Sigmoid function">sigmoid</a> loss, and the <a href="Dirac_delta_function" title="Dirac delta function">Dirac delta symbol</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \delta _{i,j}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \delta _{i,j}}</annotation>

</semantics>

</math><img src="./40e421f3ef0893f646c999fcc309a25ad6bad1f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.967ex; height:3.009ex;" alt="{\displaystyle \delta _{i,j}}" loading="lazy"> is 1 if <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle i=j}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle i=j}</annotation>

</semantics>

</math><img src="./706e0928b2bf0f24076b0c90bb20616ff2068343.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:4.859ex; height:2.509ex;" alt="{\displaystyle i=j}" loading="lazy"> else 0.

<div class="mw-heading mw-heading2"><h2 id="CLIP_models">CLIP models</h2></div>

While the original model was developed by <a href="OpenAI" title="OpenAI">OpenAI</a>, subsequent models have been trained by other organizations as well.

<div class="mw-heading mw-heading3"><h3 id="Image_model">Image model</h3></div>

The image encoding models used in CLIP are typically <a href="Vision_transformer" title="Vision transformer">vision transformers (ViT)</a>. The naming convention for these models often reflects the specific ViT architecture used. For instance, "ViT-L/14" means a "vision transformer large" (compared to other models in the same series) with a patch size of 14, meaning that the image is divided into 14-by-14 pixel patches before being processed by the transformer. The size indicator ranges from B, L, H, G (base, large, huge, giant), in that order.

Other than ViT, the image model is typically a <a href="Convolutional_neural_network" title="Convolutional neural network">convolutional neural network</a>, such as <a href="Residual_neural_network" title="Residual neural network">ResNet</a> (in the original series by OpenAI), or ConvNeXt<a href="#cite_note-7">[7]</a> (in the OpenCLIP model series by LAION<a href="#cite_note-Ilharco-8">[8]</a>).

Since the output vectors of the image model and the text model must have exactly the same length, both the image model and the text model have fixed-length vector outputs, which in the original report is called "embedding dimension".<a href="#cite_note-9">[note 1]</a>

For example, in the original OpenAI model, the ResNet models have embedding dimensions ranging from 512 to 1024,<a href="#cite_note-:1-10">[9]</a>: Table 19 and for the ViTs, from 512 to 768.<a href="#cite_note-:1-10">[9]</a>: Table 20

<caption>Models released by OpenAI<a href="#cite_note-11">[10]</a><a href="#cite_note-12">[note 2]</a>

</caption>

<th>Model name

</th>

<th>Resolution

</th>

<th>Parameters (total, in millions)

</th>

<th>Parameters (vision)

</th>

<th>Parameters (text)

</th>

<th>Embedding dimension

</th>

<th>Size (MB)

</th>

<th>Release date

</th></tr>

<tr>

<td>RN50

</td>

<td>224

</td>

<td>102

</td>

<td>38.3

</td>

<td>63.1

</td>

<td>1024

</td>

<td>244

</td>

<td>2021-01

</td></tr>

<tr>

<td>RN101

</td>

<td>224

</td>

<td>120

</td>

<td>56.3

</td>

<td>63.1

</td>

<td>512

</td>

<td>278

</td>

<td>2021-03

</td></tr>

<tr>

<td>RN50x4

</td>

<td>288

</td>

<td>178

</td>

<td>87.1

</td>

<td>90.7

</td>

<td>640

</td>

<td>402

</td>

<td>2021-03

</td></tr>

<tr>

<td>RN50x16

</td>

<td>384

</td>

<td>291

</td>

<td>167.3

</td>

<td>123.0

</td>

<td>768

</td>

<td>630

</td>

<td>2021-07

</td></tr>

<tr>

<td>RN50x64

</td>

<td>448

</td>

<td>623

</td>

<td>420.4

</td>

<td>201.8

</td>

<td>1024

</td>

<td>1260

</td>

<td>2022-01

</td></tr>

<tr>

<td>ViT-B/32

</td>

<td>224

</td>

<td>151

</td>

<td>87.8

</td>

<td>63.1

</td>

<td>512

</td>

<td>338

</td>

<td>2021-01

</td></tr>

<tr>

<td>ViT-B/16

</td>

<td>224

</td>

<td>150

</td>

<td>86.2

</td>

<td>63.1

</td>

<td>512

</td>

<td>335

</td>

<td>2021-07

</td></tr>

<tr>

<td>ViT-L/14

</td>

<td>224

</td>

<td>428

</td>

<td>304.0

</td>

<td>123.0

</td>

<td>768

</td>

<td>890

</td>

<td>2022-01

</td></tr>

<tr>

<td>ViT-L/14@336px

</td>

<td>336

</td>

<td>428

</td>

<td>304.3

</td>

<td>123.0

</td>

<td>768

</td>

<td>891

</td>

<td>2022-04

</td></tr></tbody></table>

Its implementation of ViT was the same as the original one,<a href="#cite_note-:3-13">[11]</a> with one modification: after position embeddings are added to the initial patch embeddings, there is a <a href="LayerNorm" class="mw-redirect" title="LayerNorm">LayerNorm</a>.

Its implementation of <a href="Residual_neural_network" title="Residual neural network">ResNet</a> was the same as the original one,<a href="#cite_note-resnet-14">[12]</a> with 3 modifications:

<ul><li>In the start of the CNN (the "stem"), they used three stacked 3x3 convolutions instead of a single 7x7 convolution, as suggested by.<a href="#cite_note-15">[13]</a></li>

<li>There is an average pooling of stride 2 at the start of each downsampling convolutional layer (they called it rect-2 blur pooling according to the terminology of <a href="#cite_note-16">[14]</a>). This has the effect of blurring images before downsampling, for antialiasing.<a href="#cite_note-17">[15]</a></li>

<li>The final convolutional layer is followed by a <a href="Pooling_layer#Vision_Transformer_pooling" title="Pooling layer">multiheaded attention pooling</a>.</li></ul>

ALIGN a model with similar capabilities, trained by researchers from Google<a href="#cite_note-:2-18">[16]</a> used <a href="EfficientNet" title="EfficientNet">EfficientNet</a>,<a href="#cite_note-19">[17]</a> a kind of <a href="Convolutional_neural_network" title="Convolutional neural network">convolutional neural network</a>.

<div class="mw-heading mw-heading3"><h3 id="Text_model">Text model</h3></div>

The text encoding models used in CLIP are typically <a href="Transformer_(deep_learning_architecture)" title="Transformer (deep learning architecture)">Transformers</a>.

In the original OpenAI report, they reported using a Transformer (63M-parameter, 12-layer, 512-wide, 8 attention heads) with lower-cased <a href="Byte_pair_encoding" class="mw-redirect" title="Byte pair encoding">byte pair encoding</a> (BPE) with 49152 vocabulary size. Context length was capped at 76 for efficiency. Like <a href="Generative_pre-trained_transformer" title="Generative pre-trained transformer">GPT</a>, it was decoder-only, with only causally-masked self-attention.<a href="#cite_note-:0-1">[1]</a>: <span title="Page: 5

Quotation: "Masked self-attention was used in the text encoder to preserve the ability to initialize with a pre-trained language model or add language modeling as an auxiliary objective, though exploration of this is left as future work."" class="tooltip tooltip-dashed" style="border-bottom: 1px dashed;">5 Its architecture is the same as <a href="GPT-2" title="GPT-2">GPT-2</a>.<a href="#cite_note-20">[18]</a>

Like <a href="BERT_(language_model)" title="BERT (language model)">BERT</a>, the text sequence is bracketed by two special tokens <code>[SOS]</code> and <code>[EOS]</code> ("start of sequence" and "end of sequence"). Take the activations of the highest layer of the transformer on the <code>[EOS]</code>, apply <a href="LayerNorm" class="mw-redirect" title="LayerNorm">LayerNorm</a>, then a final linear map. This is the text encoding of the input sequence. The final linear map has output dimension equal to the embedding dimension of whatever image encoder it is paired with. These models all had context length 77 and vocabulary size 49408.

ALIGN<a href="#cite_note-:2-18">[16]</a> used BERT of various sizes.

<div class="mw-heading mw-heading2"><h2 id="Dataset">Dataset</h2></div>

<div class="mw-heading mw-heading3"><h3 id="WebImageText">WebImageText</h3></div>

The CLIP models released by OpenAI were trained on a dataset called "WebImageText" (WIT) containing 400 million pairs of images and their corresponding captions scraped from the internet. The total number of words in this dataset is similar in scale to the WebText dataset used for training <a href="GPT-2" title="GPT-2">GPT-2</a>, which contains about 40 gigabytes of text data.<a href="#cite_note-:0-1">[1]</a>

The dataset contains 500,000 text-queries, with up to 20,000 (image, text) pairs per query. The text-queries were generated by starting with all words occurring at least 100 times in <a href="English_Wikipedia" title="English Wikipedia">English Wikipedia</a>, then extended by <a href="Bigram" title="Bigram">bigrams</a> with high <a href="Mutual_information" title="Mutual information">mutual information</a>, names of all Wikipedia articles above a certain search volume, and <a href="WordNet" title="WordNet">WordNet</a> <a href="Synset" title="Synset">synsets</a>.

The dataset is private and has not been released to the public, and there is no further information on it.<a href="#cite_note-22">[note 3]</a>

<div class="mw-heading mw-heading4"><h4 id="Data_preprocessing">Data preprocessing</h4></div>

For the CLIP image models, the input images are preprocessed by first dividing each of the R, G, B values of an image by the maximum possible value, so that these values fall between 0 and 1, then subtracting by <code>[0.48145466, 0.4578275, 0.40821073]</code>, and dividing by <code>[0.26862954, 0.26130258, 0.27577711]</code>.

The rationale was that these are the mean and standard deviations of the images in the WebImageText dataset, so this preprocessing step roughly <a href="Whitening_transformation" title="Whitening transformation">whitens</a> the image tensor. These numbers slightly differ from the standard preprocessing for ImageNet, which uses <code>[0.485, 0.456, 0.406]</code> and <code>[0.229, 0.224, 0.225]</code>.<a href="#cite_note-23">[20]</a>

If the input image does not have the same resolution as the native resolution (224×224 for all except ViT-L/14@336px, which has 336×336 resolution), then the input image is scaled down by <a href="Bicubic_interpolation" title="Bicubic interpolation">bicubic interpolation</a>, so that its shorter side is the same as the native resolution, then the central square of the image is <a href="Cropping_(image)" title="Cropping (image)">cropped</a> out.

<div class="mw-heading mw-heading3"><h3 id="Others">Others</h3></div>

ALIGN<a href="#cite_note-:2-18">[16]</a> used over one billion image-text pairs, obtained by extracting images and their <a href="Alt_attribute" title="Alt attribute">alt-tags</a> from online crawling. The method was described as similar to how the Conceptual Captions dataset<a href="#cite_note-24">[21]</a> was constructed, but instead of complex filtering, they only applied a frequency-based filtering.

Later models trained by other organizations had published datasets. For example, <a href="LAION" title="LAION">LAION</a> trained OpenCLIP with published datasets LAION-400M, LAION-2B, and DataComp-1B.<a href="#cite_note-25">[22]</a><a href="#cite_note-Ilharco-8">[8]</a>

<div class="mw-heading mw-heading2"><h2 id="Training">Training</h2></div>

In the original OpenAI CLIP report, they reported training 5 <a href="Residual_neural_network" title="Residual neural network">ResNet</a> and 3 <a href="Vision_transformer" title="Vision transformer">ViT</a> (ViT-B/32, ViT-B/16, ViT-L/14). Each was trained for 32 epochs. The largest ResNet model took 18 days to train on 592 <a href="Volta_(microarchitecture)#Products" title="Volta (microarchitecture)">V100</a> GPUs. The largest ViT model took 12 days on 256 V100 GPUs.

All ViT models were trained on 224×224 image resolution. The ViT-L/14 was then boosted to 336×336 resolution by FixRes,<a href="#cite_note-26">[23]</a> resulting in a model.<a href="#cite_note-27">[note 4]</a> They found this was the best-performing model.<a href="#cite_note-:0-1">[1]</a>: Appendix F. Model Hyperparameters

In the OpenCLIP series, the ViT-L/14 model was trained on 384 <a href="Ampere_(microarchitecture)" title="Ampere (microarchitecture)">A100</a> GPUs on the LAION-2B dataset, for 160 epochs for a total of 32B samples seen.<a href="#cite_note-28">[24]</a>

<div class="mw-heading mw-heading2"><h2 id="Applications">Applications</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Cross-modal_retrieval">Cross-modal retrieval</h3></div>

CLIP's cross-modal retrieval enables the alignment of visual and textual data in a shared latent space, allowing users to retrieve images based on text descriptions and vice versa, without the need for explicit image annotations.<a href="#cite_note-29">[25]</a> In text-to-image retrieval, users input descriptive text, and CLIP retrieves images with matching embeddings. In image-to-text retrieval, images are used to find related text content.

CLIP’s ability to connect visual and textual data has found applications in multimedia search, content discovery, and recommendation systems.<a href="#cite_note-30">[26]</a><a href="#cite_note-31">[27]</a>

<div class="mw-heading mw-heading3"><h3 id="Image_classification">Image classification</h3></div>

CLIP can perform zero-shot image classification tasks. This is achieved by prompting the text encoder with class names and selecting the class whose embedding is closest to the image embedding. For example, to classify an image, they compared the embedding of the image with the embedding of the text "A photo of a {class}.", and the {class} that results in the highest dot product is outputted.

<div class="mw-heading mw-heading3"><h3 id="CLIP_for_multimodal_learning">CLIP for multimodal learning</h3></div>

CLIP has been used as a component in <a href="Multimodal_learning" title="Multimodal learning">multimodal learning</a>. For example, during the training of <a href="Google_DeepMind" title="Google DeepMind">Google DeepMind</a>'s Flamingo (2022),<a href="#cite_note-32">[28]</a> the authors trained a CLIP pair, with BERT as the text encoder and NormalizerFree ResNet F6<a href="#cite_note-33">[29]</a> as the image encoder. The image encoder of the CLIP pair was taken with parameters frozen and the text encoder was discarded. The frozen image encoder was then combined with a frozen <a href="Chinchilla_(language_model)" title="Chinchilla (language model)">Chinchilla language model</a>, by finetuning with some further parameters that connect the two frozen models.

<div class="mw-heading mw-heading3"><h3 id="Applications_in_other_domains">Applications in other domains</h3></div>

<ul><li>CLIP's image encoder is a pre-trained image <a href="Feature_learning" title="Feature learning">featurizer</a>. This can then be fed into other AI models.<a href="#cite_note-:0-1">[1]</a></li></ul>

<ul><li>Models like <a href="Stable_Diffusion" title="Stable Diffusion">Stable Diffusion</a> use CLIP's text encoder to transform text prompts into embeddings for image generation.<a href="#cite_note-stable-diffusion-github-3">[3]</a> CLIP can also be used as a gradient signal for directly guiding diffusion ("CLIP guidance")<a href="#cite_note-34">[30]</a><a href="#cite_note-35">[31]</a> or other generative art.<a href="#cite_note-36">[32]</a></li></ul>

<ul><li>Fine-tuned CLIP models can be used to rank images by aesthetic quality, which may be useful as a step in filtering a large dataset into a smaller one with higher quality.<a href="#cite_note-37">[33]</a></li></ul>

<ul><li>CLIP can be used to <a href="Image_captioning" class="mw-redirect" title="Image captioning">generate image captions</a> by matching text inputs to image embeddings.<a href="#cite_note-38">[34]</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="Notes">Notes</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

<li id="cite_note-9"><a href="#cite_ref-9">^</a> Similar to the "embedding dimension" of text embedding in Transformer models.

</li>

<li id="cite_note-12"><a href="#cite_ref-12">^</a> <div class="mw-highlight mw-highlight-lang-python mw-content-ltr" dir="ltr"><pre>!pip install git+https://github.com/openai/CLIP.git

!wget https://github.com/openai/CLIP/raw/main/CLIP.png -O CLIP.png

import torch

import clip

from PIL import Image

import numpy as np

device = "cuda" if torch.cuda.is_available() else "cpu"

for m in clip.available_models():

model, preprocess = clip.load(m, device=device)

input_resolution = model.visual.input_resolution

context_length = model.context_length

vocab_size = model.vocab_size

print("Model parameters:", f"{np.sum([int(np.prod(p.shape)) for p in model.parameters()]):,}")

print("Input resolution:", input_resolution)

print("Context length:", context_length)

print("Vocab size:", vocab_size)

n_params_vision = sum(p.numel() for p in model.visual.parameters())

n_params_text = sum(p.numel() for p in model.transformer.parameters())

image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)

image_features = model.encode_image(image)

print(f"Model: {m}, #vision parameters: {n_params_vision:,}, #text parameters: {n_params_text:,}, embedding dimension: {image_features.shape[1]}")

del model, preprocess, image, image_features

</pre></div>

</li>

<li id="cite_note-22"><a href="#cite_ref-22">^</a> It is not the same as the Wikipedia-based Image Text dataset, also called "WIT".<a href="#cite_note-21">[19]</a>

</li>

<li id="cite_note-27"><a href="#cite_ref-27">^</a> They referred to this as both <code>ViT-L/14-336px</code> and <code>ViT-L/14@336px</code>, inconsistently throughout the report.

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFRadfordKimHallacyRamesh2021" class="citation conference cs1">Radford, Alec; Kim, Jong Wook; Hallacy, Chris; Ramesh, Aditya; Goh, Gabriel; Agarwal, Sandhini; Sastry, Girish; Askell, Amanda; Mishkin, Pamela; Clark, Jack; Krueger, Gretchen; Sutskever, Ilya (2021-07-01). <a rel="nofollow" class="external text" href="https://proceedings.mlr.press/v139/radford21a">Learning Transferable Visual Models From Natural Language Supervision</a>. Proceedings of the 38th International Conference on Machine Learning. PMLR. pp. 8748–8763.</cite>

</li>

<li id="cite_note-2"><a href="#cite_ref-2">^</a> <cite id="CITEREFHendriksenBleekerVakulenkovan_Noord2022" class="citation journal cs1">Hendriksen, Mariya; Bleeker, Maurits; Vakulenko, Svitlana; van Noord, Nanne; Kuiper, Ernst; de Rijke, Maarten (2022). Hagen, Matthias; Verberne, Suzan; Macdonald, Craig; Seifert, Christin; Balog, Krisztian; Nørvåg, Kjetil; Setty, Vinay (eds.). <a rel="nofollow" class="external text" href="https://link.springer.com/chapter/10.1007/978-3-030-99736-6_20">"Extending CLIP for Category-to-Image Retrieval in E-Commerce"</a>. Advances in Information Retrieval. Cham: Springer International Publishing: 289–303. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2F978-3-030-99736-6_20">10.1007/978-3-030-99736-6_20</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-3-030-99736-6</bdi>.</cite>

</li>

<li id="cite_note-stable-diffusion-github-3">^ <a href="#cite_ref-stable-diffusion-github_3-0">a</a> <a href="#cite_ref-stable-diffusion-github_3-1">b</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://github.com/CompVis/stable-diffusion">"Stable Diffusion Repository on GitHub"</a>. CompVis - Machine Vision and Learning Research Group, LMU Munich. 17 September 2022. <a rel="nofollow" class="external text" href="https://web.archive.org/web/20230118183342/https://github.com/CompVis/stable-diffusion">Archived</a> from the original on January 18, 2023. Retrieved 17 September 2022.</cite>

</li>

<li id="cite_note-4"><a href="#cite_ref-4">^</a> <cite class="citation cs2"><a rel="nofollow" class="external text" href="https://github.com/LAION-AI/aesthetic-predictor">LAION-AI/aesthetic-predictor</a>, LAION AI, 2024-09-06, retrieved 2024-09-08</cite>

</li>

<li id="cite_note-5"><a href="#cite_ref-5">^</a> <cite id="CITEREFSohn2016" class="citation journal cs1">Sohn, Kihyuk (2016). <a rel="nofollow" class="external text" href="https://proceedings.neurips.cc/paper/2016/hash/6b180037abbebea991d8b1232f8a8ca9-Abstract.html">"Improved Deep Metric Learning with Multi-class N-pair Loss Objective"</a>. Advances in Neural Information Processing Systems. 29. Curran Associates, Inc.</cite>

</li>

<li id="cite_note-:4-6"><a href="#cite_ref-:4_6-0">^</a> <cite id="CITEREFZhaiMustafaKolesnikovBeyer2023" class="citation conference cs1">Zhai, Xiaohua; Mustafa, Basil; Kolesnikov, Alexander; Beyer, Lucas (2023). <a rel="nofollow" class="external text" href="https://openaccess.thecvf.com/content/ICCV2023/html/Zhai_Sigmoid_Loss_for_Language_Image_Pre-Training_ICCV_2023_paper.html">Sigmoid Loss for Language Image Pre-Training</a>. IEEE/CVF International Conference on Computer Vision (ICCV). pp. 11975–11986.</cite>

</li>

<li id="cite_note-7"><a href="#cite_ref-7">^</a> <cite id="CITEREFLiuMaoWuFeichtenhofer2022" class="citation conference cs1">Liu, Zhuang; Mao, Hanzi; Wu, Chao-Yuan; Feichtenhofer, Christoph; Darrell, Trevor; Xie, Saining (2022). <a rel="nofollow" class="external text" href="https://openaccess.thecvf.com/content/CVPR2022/html/Liu_A_ConvNet_for_the_2020s_CVPR_2022_paper.html">A ConvNet for the 2020s</a>. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11976–11986.</cite>

</li>

<li id="cite_note-Ilharco-8">^ <a href="#cite_ref-Ilharco_8-0">a</a> <a href="#cite_ref-Ilharco_8-1">b</a> <cite id="CITEREFIlharcoWortsmanWightmanGordon2021" class="citation cs2">Ilharco, Gabriel; Wortsman, Mitchell; Wightman, Ross; Gordon, Cade; <a href="Nicholas_Carlini" title="Nicholas Carlini">Carlini, Nicholas</a>; Taori, Rohan; Dave, Achal; Shankar, Vaishaal; Namkoong, Hongseok (July 2021), <a rel="nofollow" class="external text" href="https://github.com/mlfoundations/open_clip">OpenCLIP</a>, <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.5281%2Fzenodo.5143773">10.5281/zenodo.5143773</a>, retrieved 2024-09-06</cite>

</li>

<li id="cite_note-:1-10">^ <a href="#cite_ref-:1_10-0">a</a> <a href="#cite_ref-:1_10-1">b</a> <cite id="CITEREFRadfordKimHallacyRamesh2021" class="citation arxiv cs1">Radford, Alec; Kim, Jong Wook; Hallacy, Chris; Ramesh, Aditya; Goh, Gabriel; Agarwal, Sandhini; Sastry, Girish; Askell, Amanda; Mishkin, Pamela; Clark, Jack; Krueger, Gretchen; Sutskever, Ilya (2021). "Learning Transferable Visual Models From Natural Language Supervision". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2103.00020">2103.00020</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CV">cs.CV</a>].</cite>

</li>

<li id="cite_note-11"><a href="#cite_ref-11">^</a> <cite class="citation cs2"><a rel="nofollow" class="external text" href="https://github.com/openai/CLIP/">openai/CLIP</a>, OpenAI, 2024-09-06, retrieved 2024-09-06</cite>

</li>

<li id="cite_note-:3-13"><a href="#cite_ref-:3_13-0">^</a> <cite id="CITEREFDosovitskiyBeyerKolesnikovWeissenborn2021" class="citation arxiv cs1">Dosovitskiy, Alexey; Beyer, Lucas; Kolesnikov, Alexander; Weissenborn, Dirk; Zhai, Xiaohua; Unterthiner, Thomas; Dehghani, Mostafa; Minderer, Matthias; Heigold, Georg; Gelly, Sylvain; Uszkoreit, Jakob (2021-06-03). "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2010.11929">2010.11929</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CV">cs.CV</a>].</cite>

</li>

<li id="cite_note-resnet-14"><a href="#cite_ref-resnet_14-0">^</a> <cite id="CITEREFHeZhangRenSun2015" class="citation conference cs1">He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (10 Dec 2015). Deep Residual Learning for Image Recognition. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1512.03385">1512.03385</a>.</cite>

</li>

<li id="cite_note-15"><a href="#cite_ref-15">^</a> <cite id="CITEREFHeZhangZhangZhang2018" class="citation arxiv cs1">He, Tong; Zhang, Zhi; Zhang, Hang; Zhang, Zhongyue; Xie, Junyuan; Li, Mu (2018-12-05). "Bag of Tricks for Image Classification with Convolutional Neural Networks". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1812.01187">1812.01187</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CV">cs.CV</a>].</cite>

</li>

<li id="cite_note-16"><a href="#cite_ref-16">^</a> <cite id="CITEREFZhang2018" class="citation journal cs1">Zhang, Richard (2018-09-27). <a rel="nofollow" class="external text" href="https://openreview.net/forum?id=SklVEnR5K7">"Making Convolutional Networks Shift-Invariant Again"</a>.</cite> <code class="cs1-code">{{cite journal}}</code>: Cite journal requires <code class="cs1-code">|journal=</code> (help)

</li>

<li id="cite_note-17"><a href="#cite_ref-17">^</a> <cite id="CITEREFZhang2019" class="citation arxiv cs1">Zhang, Richard (2019-06-08). "Making Convolutional Networks Shift-Invariant Again". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1904.11486">1904.11486</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CV">cs.CV</a>].</cite>

</li>

<li id="cite_note-:2-18">^ <a href="#cite_ref-:2_18-0">a</a> <a href="#cite_ref-:2_18-1">b</a> <a href="#cite_ref-:2_18-2">c</a> <cite id="CITEREFJiaYangXiaChen2021" class="citation journal cs1">Jia, Chao; Yang, Yinfei; Xia, Ye; Chen, Yi-Ting; Parekh, Zarana; Pham, Hieu; Le, Quoc; Sung, Yun-Hsuan; Li, Zhen; Duerig, Tom (2021-07-01). <a rel="nofollow" class="external text" href="https://proceedings.mlr.press/v139/jia21b.html">"Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision"</a>. Proceedings of the 38th International Conference on Machine Learning. PMLR: 4904–4916.</cite>

</li>

<li id="cite_note-19"><a href="#cite_ref-19">^</a> <cite id="CITEREFTanLe2020" class="citation arxiv cs1">Tan, Mingxing; Le, Quoc V. (2020-09-11). "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1905.11946">1905.11946</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-20"><a href="#cite_ref-20">^</a> <cite id="CITEREFRadfordWuChildLuan2019" class="citation journal cs1">Radford, Alec; Wu, Jeff; Child, R.; Luan, D.; Amodei, Dario; Sutskever, I. (2019). "Language Models are Unsupervised Multitask Learners". <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:160025533">160025533</a>.</cite> <code class="cs1-code">{{cite journal}}</code>: Cite journal requires <code class="cs1-code">|journal=</code> (help)

</li>

<li id="cite_note-21"><a href="#cite_ref-21">^</a> <cite id="CITEREFSrinivasanRamanChenBendersky2021" class="citation book cs1">Srinivasan, Krishna; Raman, Karthik; Chen, Jiecao; Bendersky, Michael; Najork, Marc (2021-07-11). "WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning". Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2443–2449. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2103.01913">2103.01913</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1145%2F3404835.3463257">10.1145/3404835.3463257</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-4503-8037-9</bdi>.</cite>

</li>

<li id="cite_note-23"><a href="#cite_ref-23">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://github.com/openai/CLIP/issues/20">"std and mean for image normalization different from ImageNet · Issue #20 · openai/CLIP"</a>. GitHub. Retrieved 2024-09-19.</cite>

</li>

<li id="cite_note-24"><a href="#cite_ref-24">^</a> <cite id="CITEREFSharmaDingGoodmanSoricut2018" class="citation journal cs1">Sharma, Piyush; Ding, Nan; Goodman, Sebastian; Soricut, Radu (July 2018). Gurevych, Iryna; Miyao, Yusuke (eds.). <a rel="nofollow" class="external text" href="https://aclanthology.org/P18-1238/">"Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning"</a>. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics: 2556–2565. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.18653%2Fv1%2FP18-1238">10.18653/v1/P18-1238</a>.</cite>

</li>

<li id="cite_note-25"><a href="#cite_ref-25">^</a> <cite id="CITEREFChertiBeaumontWightmanWortsman2023" class="citation book cs1">Cherti, Mehdi; Beaumont, Romain; Wightman, Ross; Wortsman, Mitchell; Ilharco, Gabriel; Gordon, Cade; Schuhmann, Christoph; Schmidt, Ludwig; Jitsev, Jenia (June 2023). "Reproducible Scaling Laws for Contrastive Language-Image Learning". 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2818–2829. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2212.07143">2212.07143</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FCVPR52729.2023.00276">10.1109/CVPR52729.2023.00276</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>979-8-3503-0129-8</bdi>.</cite>

</li>

<li id="cite_note-26"><a href="#cite_ref-26">^</a> <cite id="CITEREFTouvronVedaldiDouzeJegou2019" class="citation journal cs1">Touvron, Hugo; Vedaldi, Andrea; Douze, Matthijs; Jegou, Herve (2019). <a rel="nofollow" class="external text" href="https://proceedings.neurips.cc/paper/2019/hash/d03a857a23b5285736c4d55e0bb067c8-Abstract.html">"Fixing the train-test resolution discrepancy"</a>. Advances in Neural Information Processing Systems. 32. Curran Associates, Inc.</cite>

</li>

<li id="cite_note-28"><a href="#cite_ref-28">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K">"laion/CLIP-ViT-L-14-laion2B-s32B-b82K · Hugging Face"</a>. huggingface.co. 2023-09-10. Retrieved 2024-09-06.</cite>

</li>

<li id="cite_note-29"><a href="#cite_ref-29">^</a> <cite id="CITEREFHendriksenBleekerVakulenkovan_Noord2021" class="citation arxiv cs1">Hendriksen, Mariya; Bleeker, Maurits; Vakulenko, Svitlana; van Noord, Nanne; Kuiper, Ernst; de Rijke, Maarten (2021). "Extending CLIP for Category-to-image Retrieval in E-commerce". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2112.11294">2112.11294</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CV">cs.CV</a>].</cite>

</li>

<li id="cite_note-30"><a href="#cite_ref-30">^</a> <cite id="CITEREFBeaumont2024" class="citation cs2">Beaumont, Romain (2024-09-07), <a rel="nofollow" class="external text" href="https://github.com/rom1504/clip-retrieval">rom1504/clip-retrieval</a>, retrieved 2024-09-08</cite>

</li>

<li id="cite_note-31"><a href="#cite_ref-31">^</a> <cite id="CITEREFHaltakov2024" class="citation cs2">Haltakov, Vladimir (2024-09-03), <a rel="nofollow" class="external text" href="https://github.com/haltakov/natural-language-image-search">haltakov/natural-language-image-search</a>, retrieved 2024-09-06</cite>

</li>

<li id="cite_note-32"><a href="#cite_ref-32">^</a> <cite id="CITEREFAlayracDonahueLucMiech2022" class="citation journal cs1">Alayrac, Jean-Baptiste; Donahue, Jeff; Luc, Pauline; Miech, Antoine; Barr, Iain; Hasson, Yana; Lenc, Karel; Mensch, Arthur; Millican, Katherine; Reynolds, Malcolm; Ring, Roman; Rutherford, Eliza; Cabi, Serkan; Han, Tengda; Gong, Zhitao (2022-12-06). <a rel="nofollow" class="external text" href="https://proceedings.neurips.cc/paper_files/paper/2022/hash/960a172bc7fbf0177ccccbb411a7d800-Abstract-Conference.html">"Flamingo: a Visual Language Model for Few-Shot Learning"</a>. Advances in Neural Information Processing Systems. 35: 23716–23736.</cite>

</li>

<li id="cite_note-33"><a href="#cite_ref-33">^</a> <cite id="CITEREFBrockDeSmithSimonyan2021" class="citation journal cs1">Brock, Andy; De, Soham; Smith, Samuel L.; Simonyan, Karen (2021-07-01). <a rel="nofollow" class="external text" href="https://proceedings.mlr.press/v139/brock21a.html">"High-Performance Large-Scale Image Recognition Without Normalization"</a>. Proceedings of the 38th International Conference on Machine Learning. PMLR: 1059–1071.</cite>

</li>

<li id="cite_note-34"><a href="#cite_ref-34">^</a> <cite id="CITEREFRameshDhariwalNicholChu2022" class="citation arxiv cs1">Ramesh, Aditya; Dhariwal, Prafulla; Nichol, Alex; Chu, Casey; Chen, Mark (2022-04-12). "Hierarchical Text-Conditional Image Generation with CLIP Latents". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2204.06125">2204.06125</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CV">cs.CV</a>].</cite>

</li>

<li id="cite_note-35"><a href="#cite_ref-35">^</a> <cite id="CITEREFHendriksenBleekerVakulenkovan_Noord2022" class="citation journal cs1">Hendriksen, Mariya; Bleeker, Maurits; Vakulenko, Svitlana; van Noord, Nanne; Kuiper, Ernst; de Rijke, Maarten (2022). Hagen, Matthias; Verberne, Suzan; Macdonald, Craig; Seifert, Christin; Balog, Krisztian; Nørvåg, Kjetil; Setty, Vinay (eds.). <a rel="nofollow" class="external text" href="https://link.springer.com/chapter/10.1007/978-3-030-99736-6_20">"Extending CLIP for Category-to-Image Retrieval in E-Commerce"</a>. Advances in Information Retrieval. Cham: Springer International Publishing: 289–303. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2F978-3-030-99736-6_20">10.1007/978-3-030-99736-6_20</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-3-030-99736-6</bdi>.</cite>

</li>

<li id="cite_note-36"><a href="#cite_ref-36">^</a> <cite id="CITEREFWhitaker2022" class="citation web cs1">Whitaker, Jonathan (2022-05-22). <a rel="nofollow" class="external text" href="https://wandb.ai/johnowhitaker/nca/reports/Fun-with-Neural-Cellular-Automata--VmlldzoyMDQ5Mjg0">"Fun With Neural Cellular Automata"</a>. W&B. Retrieved 2024-09-08.</cite>

</li>

<li id="cite_note-37"><a href="#cite_ref-37">^</a> <cite class="citation cs2"><a rel="nofollow" class="external text" href="https://github.com/LAION-AI/aesthetic-predictor">LAION-AI/aesthetic-predictor</a>, LAION AI, 2024-09-06, retrieved 2024-09-08</cite>

</li>

<li id="cite_note-38"><a href="#cite_ref-38">^</a> <cite id="CITEREFMokadyHertzBermano2021" class="citation arxiv cs1">Mokady, Ron; Hertz, Amir; Bermano, Amit H. (2021). "ClipCap: CLIP Prefix for Image Captioning". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2111.09734">2111.09734</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CV">cs.CV</a>].</cite>

</li>

</ol></div>

<div class="mw-heading mw-heading2"><h2 id="External_links">External links</h2></div>

<ul><li><a rel="nofollow" class="external text" href="https://openai.com/research/clip">OpenAI's CLIP webpage</a></li>

<li><a rel="nofollow" class="external text" href="https://github.com/mlfoundations/open_clip">OpenCLIP: An open source implementation of CLIP</a></li>

<li><cite id="CITEREFArora2023" class="citation web cs1">Arora, Aman (2023-03-11). <a rel="nofollow" class="external text" href="https://amaarora.github.io/posts/2023-03-11_Understanding_CLIP_part_2.html">"The Annotated CLIP (Part-2)"</a>. amaarora.github.io. Retrieved 2024-09-11.</cite></li></ul></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-06-21" href="https://en.wikipedia.org/wiki/?title=Contrastive_Language-Image_Pre-training&oldid=1296674956">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>